Papers with Empirical analysis
Show Your Work with Confidence: Confidence Bands for Tuning Curves (2024.naacl-long)
Copied to clipboard
| Challenge: | a rush to scale up has left us with large, costly language models and little understanding of how different designs compare. |
| Approach: | They propose a method to construct valid confidence bands for tuning curves . they validated their method with ablations and analyze the effect of sample size . |
| Outcome: | The proposed method shows that bootstrap confidence bands do not approximate their target confidence. |
SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training (2024.acl-long)
Copied to clipboard
| Challenge: | Current methods focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. |
| Approach: | They propose a method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. |
| Outcome: | The proposed method significantly improves training efficiency on deduplicated datasets and improves downstream accuracy by 1.77%. |
Learning to Ask for Conversational Machine Learning (D19-1)
Copied to clipboard
| Challenge: | Empirical analysis across three domains shows that learned question-asking strategies expedite classifier training by asking appropriate questions at different points in the learning process. |
| Approach: | They propose a reinforcement learning framework where the learner’s actions correspond to question types and the reward for asking a question is based on how the teacher’s response changes performance of the resulting machine learning model. |
| Outcome: | The proposed framework outperforms a random policy on learning classification tasks, but the dialog looks contrived from a human perspective. |
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain QA tasks focus on questions whose answer can be deduced directly from global factual knowledge. |
| Approach: | They propose a dataset where each question is based on a counterfactual presupposition via an "if" clause. |
| Outcome: | The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. |
Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking (2025.acl-long)
Copied to clipboard
Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang
| Challenge: | Large language models face inherent performance bottlenecks under parameter constraints . challenging tokens induce abrupt gradient spikes across layers, exposing stress points . |
| Approach: | They propose an inner thinking transformer that reimagines layer computations as implicit thinking steps. |
| Outcome: | Empirical results show that ITT outperforms Transformer/Loop variants in 11 benchmarks. |